Systems and Methods for Sampling Images

ABSTRACT

An example method includes determining, by a controller of an image capture system, a plurality of sets of exposure parameter values for one or more exposure parameters. The plurality of sets of exposure parameter values are determined at an exposure determination rate. The method further includes capturing, by an image capture device of the image capture system, a plurality of images. Each image of the plurality of images is captured according to a set of exposure parameter values of the plurality of sets of exposure parameter values. The method also includes sending, by the controller of the image capture system to an image processing unit, a subset of the plurality of images. Each subset of images is sent at a sampling rate, and the sampling rate is less than the exposure determination rate.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of, and claims priority to,U.S. patent application Ser. No. No. 17/448,481, filed on Sep. 22, 2021,which is a continuation of, and claims priority to U.S. patentapplication Ser. No. 16/804,052, filed on Feb. 28, 2020, which is nowU.S. Pat. No. 11,153,501, issued on Oct. 19, 2021, which is acontinuation of, and claims priority to, U.S. patent application Ser.No. 15/851,901, filed Dec. 22, 2017, which is now U.S. Pat. No.10,609,294, issued on Mar. 31, 2020, and which are hereby incorporatedby reference into the present application in their entirety.

BACKGROUND

As technology advances, various types of robotic devices are beingcreated for performing a variety of functions that may assist users.Robotic devices may be used for applications involving materialhandling, transportation, welding, assembly, and dispensing, amongothers. Over time, the manner in which these robotic systems operate isbecoming more intelligent, efficient, and intuitive. As robotic systemsbecome increasingly prevalent in numerous aspects of modern life, it isdesirable for robotic systems to be efficient. Therefore, a demand forefficient robotic systems has helped open up a field of innovation inactuators, movement, sensing techniques, as well as component design andassembly.

SUMMARY

The present application discloses implementations that relate to imageprocessing systems. In one example, the present application describes amethod. The method includes determining, by a controller of an imagecapture system, a plurality of sets of exposure parameter values for oneor more exposure parameters. The plurality of sets of exposure parametervalues are determined at an exposure determination rate, where theexposure determination rate includes a rate at which captured images areprocessed by the controller to determine each set of exposure parametervalues. The method further includes capturing, by an image capturedevice of the image capture system, a plurality of images. Each image ofthe plurality of images is captured according to a set of exposureparameter values of the plurality of sets of exposure parameter values.The method also includes sending, by the controller of the image capturesystem to an image processing unit, a subset of the plurality of images.Each subset of images is sent at a sampling rate, and the sampling rateis less than the exposure determination rate.

In another example, the present application describes a system. Thesystem includes at least one image capture device, an image processingunit, and a controller having one or more processors. The system furtherincludes a non-transitory computer readable medium and programinstructions stored on the non-transitory computer readable medium andexecutable by the one or more processors to perform functions. Theinstructions are executable to cause the at least one image capturedevice to capture a plurality of images. The instructions are furtherexecutable to determine a plurality of sets of exposure parameter valuesfor one or more exposure parameters that correspond to the plurality ofimages. The plurality of sets of exposure parameter values aredetermined at an exposure determination rate, where the exposuredetermination rate includes a rate at which the plurality of images areprocessed by the controller to determine each set of exposure parametervalues. The instructions are also executable to send, to the imageprocessing unit, a subset of the plurality of images, wherein the subsetof images is sent at a sampling rate, and wherein the sampling rate isless than the exposure determination rate.

In yet another example, the present application describes anon-transitory computer readable medium. The non-transitory computerreadable medium has stored thereon instructions executable by one ormore processors to cause a computing system to perform functions. Thefunctions include determining, by a controller of an image capturesystem, a plurality of sets of exposure parameter values for one or moreexposure parameters. The plurality of sets of exposure parameter valuesare determined at an exposure determination rate, where the exposuredetermination rate includes a rate at which captured images areprocessed by the controller to determine each set of exposure parametervalues. The functions further include capturing, by an image capturedevice of the image capture system, a plurality of images. Each image ofthe plurality of images is captured according to a set of exposureparameter values of the plurality of sets of exposure parameter values.The functions also include sending, by the controller of the imagecapture system to an image processing unit, a subset of the plurality ofimages. Each subset of images is sent at a sampling rate, and thesampling rate is less than the exposure determination rate.

In an additional example, the present application describes a system.The system includes means for determining a plurality of sets ofexposure parameter values for one or more exposure parameters. Theplurality of sets of exposure parameter values are determined at anexposure determination rate, where the exposure determination rateincludes a rate at which captured images are processed by the controllerto determine each set of exposure parameter values. The system furtherincludes means for capturing a plurality of images. Each image of theplurality of images is captured according to a set of exposure parametervalues of the plurality of sets of exposure parameter values. The systemalso includes means for sending, to an image processing unit, a subsetof the plurality of images. Each subset of images is sent at a samplingrate, and the sampling rate is less than the exposure determinationrate.

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects, embodiments,and features described above, further aspects, embodiments, and featureswill become apparent by reference to the figures and the followingdetailed description and the accompanying drawings.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates a configuration of a robotic system, according toexample embodiments.

FIG. 2 illustrates a robotic arm, according to example embodiments.

FIG. 3 illustrates a configuration of an image processing system,according to example embodiments.

FIG. 4 illustrates another configuration of an image processing system,according to example embodiments.

FIG. 5 illustrates a timing diagram for an image processing system,according to example embodiments.

FIG. 6 illustrates operation of an image processing system, according toexample embodiments.

FIG. 7 illustrates operation of another image processing system,according to example embodiments.

FIG. 8 is a block diagram of a method, according to example embodiments.

DETAILED DESCRIPTION

The following detailed description describes various features andoperations of the disclosed systems and methods with reference to theaccompanying figures. The illustrative system and method embodimentsdescribed herein are not meant to be limiting. It may be readilyunderstood that certain aspects of the disclosed systems and methods canbe arranged and combined in a wide variety of different configurations,all of which are contemplated herein.

I. Overview

The present application discloses implementations that relate to imageprocessing systems. An example system includes an image capture systemand an image processing unit. The image capture system includes an imagecapture device and a controller. The image capture device is configuredto capture images at a capture rate, and the controller is configured todetermine a set of one or more exposure parameter values for each of thecaptured images at an exposure determination rate. The images aresampled by the controller and sent to the image processing unit at asampling rate that is less than the exposure determination rate. Theterm “sampling rate” refers to a number of images that are sent by thecontroller to the image processing unit per unit time. Accordingly, onlya subset of the captured images are processed by the image processingunit. Various types of image processing operations may be performed bythe image processing unit on the sampled images, and the sampling ratemay be determined based on an amount of time associated with performingthe operations.

Many systems use image processing systems to operate effectively. Forinstance, a robot may use an image processing system to recognizeobjects in an environment of the robot. Recognizing such objects mayallow the robot to navigate within the environment, or to interact withthe detected objects. Thus, a robot may rely on the accuracy of suchimage processing operations. Further, a robot may need to perform suchoperations quickly.

A number of factors can affect the accuracy and speed of imageprocessing operations. For example, dim photos may reduce the accuracyof color detection operations and off-focus or blurry images mayincrease the time required to perform edge detection operations. Toaccount for this, image processing systems may perform pre-processing onimages, alter parameters used when capturing images, or may only allowthe capture of images in certain conditions. One such example caninclude performing auto-exposure on images. This allows subsequentoperations to be performed more quickly and accurately because thesubsequent operations use images with desired histogram distributions.Auto-exposure operations are also computationally simple relative toother operations, such as object detection operations or debluroperations, so auto-exposure operations can be performed quickly.

Thus, various embodiments described herein include an image capturesystem that performs auto-exposure on a plurality of captured images.Each of the plurality of captured images is captured according to a setof one or more exposure parameter values determined by a controller ofthe image capture system. The controller can use pixel statistics of oneor more prior images when determining the set of exposure parametervalues for a captured image. Because determining the set of exposureparameters can be performed quickly, an image capture device of theimage capture system can capture images at a high rate, while thecontroller determines the set of exposure parameters for each capturedimage.

An image processing unit may perform image processing operations thatare more computationally taxing than an auto-exposure operation.Accordingly, such operations may take longer than it takes thecontroller to determine sets of exposure parameter values. It thereforemay be impracticable for the image processing unit to perform imageprocessing operations on each of the plurality of captured images. Butcapturing images at a high rate allows the controller to determine setsof exposure parameter values that more effectively represent the scenecaptured in the images. Accordingly, the controller can sample some, butnot all, of the images from the plurality of images captured by theimage capture device to send to the image processing unit. This allowsthe sampled images to more effectively represent a scene, while stillgiving the image processing unit time to perform operations on an imagebefore receiving another image. Further, because the controller does notsend each captured image, less communication bandwidth is used. This canallow for more efficient communications between the controller and theimage processing unit, particularly where the controller wirelesslycommunicates with the image processing unit.

In some examples, additional steps may be taken to ensure that an imagereceived by the image processing unit is fit for image processingoperations. For example, a pre-processing unit may be included withinthe system that performs intermediate processing of an image. Forexample, the pre-processing unit can sample images in parallel with theimage processing unit and determine a white level or focus level forthose images. The pre-processing unit, or the controller of the imagecapture system, can then determine which images to send to the imageprocessing unit based on the determined white level or focus level. Insome examples, the pre-processing unit and the controller of the imagecapture device can be one and the same.

In some examples, an image processing system can be part of anothersystem, such as a robotic system. The robotic system may include a robotcontroller that determines actions for the robot, which may affect theoperations performed by the image capture system. For example, the robotcontroller might determine a motion command for a robot that makes itmore likely that the scene captured by the image capture device willexperience a large change.

In some embodiments, the sampling rate is constant (e.g., the controllersamples and sends every nth captured image to the image processingunit). In other embodiments, the controller of the image capture systemcan also alter the sampling rate based on its own determinations. Forinstance, the controller can determine a large change in the scene basedon an error term calculated from pixel statistics in various images.Such an error term may be indicative of a large shift in histogram databetween two or more captured images. The controller can wait for theerror term to return to a normal level before sending an image to theimage processing unit.

In further examples, the image processing unit can also cause thecontroller to alter the sampling rate with a feedback signal. Forinstance, the image processing unit can send a signal to the controllereach time it completes an image processing operation such that thesampling rate is a variable rate consistent with an image processingoperation time for each image. The image processing unit can also changethe sampling rate based on a type of image processing operation it is toperform. For instance, the image processing unit may receive a commandsignal that instructs it to perform object detection operations for afirst period, and to perform deblur operations in addition to the objectdetection operations for a second period. The image processing unit canset a first sampling rate during the first period and set a secondsampling rate during the second period, where each sampling rate iscommensurate with an expected image processing time.

In addition to variable sampling rates based on actions of a robot,feedback from an image processing unit, feedback from a pre-processingunit, or determinations by the controller of the image capture system,various exposure parameter values can be altered based on such inputs.For example, an exposure duration or gain associated with a set ofexposure parameters can be altered based on a type of image processingoperation performed by the image processing unit, or based on a type ofaction carried out by a robot.

These and various other embodiments are described herein. It should beunderstood that, though an image processing system is described in thecontext of a larger robotic system, this system can be a standalonesystem included within a single unitary device, such as a videorecording device, a camera, or an imager. Further, the image processingsystem may be included within other types of systems such as in amanufacturing system, automotive vehicle system, or cellular handhelddevice system.

Reference will now be made in detail to various embodiments, examples ofwhich are illustrated in the accompanying drawings. In the followingdetailed description, numerous specific details are set forth in orderto provide a thorough understanding of the present disclosure and thedescribed embodiments. However, the present disclosure may be practicedwithout these specific details. In other instances, well-known methods,procedures, and components, and circuits have not been described indetail so as not to unnecessarily obscure aspects of the embodiments.

II. Example Robotic Systems

FIG. 1 illustrates an example configuration of a robotic system that maybe used in connection with the implementations described herein. Therobotic system 100 may be a robotic arm, a different type of roboticmanipulator, or it may have a number of different forms. Additionally,the robotic system 100 may also be referred to as a robotic device,robotic manipulator, or robot, among others.

The robotic system 100 is shown to include processor(s) 102, datastorage 104, program instructions 106, controller 108, sensor(s) 110,power source(s) 112, actuator(s) 114, and movable component(s) 116. Notethat the robotic system 100 is shown for illustration purposes only asrobotic system 100 may include additional components and/or have one ormore components removed without departing from the scope of theinvention. Further, note that the various components of robotic system100 may be connected in any manner.

Processor(s) 102 may be a general-purpose processor or a special purposeprocessor (e.g., digital signal processors, application specificintegrated circuits, etc.). The processor(s) 102 can be configured toexecute computer-readable program instructions 106 that are stored inthe data storage 104 and are executable to provide the functionality ofthe robotic system 100 described herein. For instance, the programinstructions 106 may be executable to provide functionality ofcontroller 108, where the controller 108 may be configured to instructan actuator 114 to cause movement of one or more movable component(s)116.

The data storage 104 may include or take the form of one or morecomputer-readable storage media that can be read or accessed byprocessor(s) 102. The one or more computer-readable storage media caninclude volatile and/or non-volatile storage components, such asoptical, magnetic, organic or other memory or disc storage, which can beintegrated in whole or in part with processor(s) 102. In someembodiments, the data storage 104 can be implemented using a singlephysical device (e.g., one optical, magnetic, organic or other memory ordisc storage unit), while in other embodiments, the data storage 104 canbe implemented using two or more physical devices. Further, in additionto the computer-readable program instructions 106, the data storage 104may include additional data such as diagnostic data, among otherpossibilities.

The robotic system 100 may include one or more sensor(s) 110 such asforce sensors, proximity sensors, motion sensors, load sensors, positionsensors, touch sensors, depth sensors, ultrasonic range sensors, andinfrared sensors, among other possibilities. The sensor(s) 110 mayprovide sensor data to the processor(s) 102 to allow for appropriateinteraction of the robotic system 100 with the environment.Additionally, the sensor data may be used in evaluation of variousfactors for providing feedback as further discussed below. Further, therobotic system 100 may also include one or more power source(s) 112configured to supply power to various components of the robotic system100. Any type of power source may be used such as, for example, agasoline engine or a battery.

The robotic system 100 may also include one or more actuator(s) 114. Anactuator is a mechanism that may be used to introduce mechanical motion.In particular, an actuator may be configured to convert stored energyinto movement of one or more components. Various mechanisms may be usedto power an actuator. For instance, actuators may be powered bychemicals, compressed air, or electricity, among other possibilities. Insome cases, an actuator may be a rotary actuator that may be used insystems involving rotational forms of motion (e.g., a joint in therobotic system 100). In other cases, an actuator may be a linearactuator that may be used in systems involving straight line motion.

In either case, actuator(s) 114 may cause movement of various movablecomponent(s) 116 of the robotic system 100. The moveable component(s)116 may include appendages such as robotic arms, legs, and/or hands,among others. The moveable component(s) 116 may also include a movablebase, wheels, and/or end effectors, among others.

In some implementations, a computing system (not shown) may be coupledto the robotic system 100 and may be configured to receive input from auser, such as via a graphical user interface. This computing system maybe incorporated within the robotic system 100 or may be an externalcomputing system that is capable of (wired or wireless) communicationwith the robotic system 100. As such, the robotic system 100 may receiveinformation and instructions, such as based on user-input at thegraphical user interface and/or based on user-input received via pressof buttons (or tactile input) on the robotic system 100, among otherpossibilities.

A robotic system 100 may take on various forms. To illustrate, FIG. 2shows an example robotic arm 200. As shown, the robotic arm 200 includesa base 202, which may be a stationary base or may be a movable base. Inthe case of a movable base, the base 202 may be considered as one of themovable component(s) 116 and may include wheels (not shown), powered byone or more of the actuator(s) 114, which allow for mobility of theentire robotic arm 200.

Additionally, the robotic arm 200 includes joints 204A-204F each coupledto one or more of the actuator(s) 114. The actuators in joints 204A-204Fmay operate to cause movement of various movable component(s) 116 suchas appendages 206A-206F and/or end effector 208. For example, theactuator in joint 204F may cause movement of appendage 206F and endeffector 208 (i.e., since end effector 208 is coupled to appendage206F). Further, end effector 208 may take on various forms and mayinclude various parts. In one example, end effector 208 may take theform of a gripper such as a finger gripper as shown here or a differenttype of gripper such as a suction gripper. In another example, endeffector 208 may take the form of a tool such as a drill or a brush. Inyet another example, the end effector may include sensors such as forcesensors, location sensors, and/or proximity sensors. Other examples mayalso be possible.

In an example implementation, a robotic system 100, such as robotic arm200, may be capable of operating in a teach mode. In particular, teachmode may be an operating mode of the robotic arm 200 that allows a userto physically interact with and guide the robotic arm 200 towardscarrying out and recording various movements. In a teaching mode, anexternal force is applied (e.g., by the user) to the robotic system 100based on a teaching input that is intended to teach the robotic systemregarding how to carry out a specific task. The robotic arm 200 may thusobtain data regarding how to carry out the specific task based oninstructions and guidance from the user. Such data may relate to aplurality of configurations of the movable component(s) 116, jointposition data, velocity data, acceleration data, torque data, forcedata, and power data, among other possibilities.

For example, during teach mode the user may grasp onto any part of therobotic arm 200 and provide an external force by physically moving therobotic arm 200. In particular, the user may guide the robotic arm 200towards grasping onto an object and then moving the object from a firstlocation to a second location. As the user guides the robotic arm 200during teach mode, the system may obtain and record data related to themovement such that the robotic arm 200 may be configured toindependently carry out the task at a future time during independentoperation (e.g., when the robotic arm 200 operates independently outsideof teach mode). Note, however, that external forces may also be appliedby other entities in the physical workspace such as by other objects,machines, and/or robotic systems, among other possibilities.

III. Example Image Processing Systems

FIG. 3 illustrates a configuration of an image processing system 300,according to example embodiments. Image processing system 300 includesan image capture system 302 having an image capture device 304 and acontroller 306. The term “image capture device” refers to an opticalinstrument for recording or capturing images. The term “controller”refers to one or more processors configured to control the operation ofthe image capture device. Image capture device 304 and controller 306are depicted as communicating back and forth. For example, controller306 can send control signals indicative of a capture rate at which imagecapture device 304 is to capture a plurality of images, while imagecapture device 304 can send the plurality of images to controller 306.Controller 306 can determine sets of one or more exposure parametervalues for image capture device 304 to use when capturing images. Forinstance, controller 306 can receive a first image, determine a setexposure parameter values based on pixel statistics of the first image,and send the set of exposure parameter values to image capture device304. The term “exposure parameter” refers to at least one of anaperture, gain, or exposure duration used by the image capture devicewhen capturing an image. The set of exposure parameter values caninclude at least one of an exposure duration value, gain value, and/oran aperture value for capturing a subsequent image. Image capture device304 can capture a second image according to the determined set ofexposure parameter values.

As controller 306 determines the sets of exposure parameter values,controller 306 can also determine error terms for each captured image.The term “error term” refers to a level of change between pixelstatistics associated with two or more captured images. For example,controller 306 can determine a level of variance between a firsthistogram associated with a first image and a second histogramassociated with a second image. A large variance may indicate a suddenchange in lighting within the environment. Though successivelydetermined sets of exposure parameter values after the second image iscaptured may adjust for this change, it may take several images forcontroller 306 to reach a stable desired histogram. Thus, controller 306may wait until the error term associated with a given image falls withinan error term threshold before sampling that image for processing byimage processing unit 308.

Image processing system 300 also includes an image processing unit 308.Image processing unit 308 can receive a subset of the plurality ofimages captured by the image capture system and perform image processingoperations on the captured images. Image processing unit 308 and imagecapture system 302 are depicted as communicating back and forth. Forinstance, controller 306 of image capture system 302 can send an imageto image capture system 308, and image capture system 308 can send afeedback signal to controller 306. The feedback signal may indicate tocontroller 306 that image capture system 308 is ready to receive anotherimage.

Controller 306 can send images to image processing unit 308 at asampling rate. In some examples, controller 306 can send images to imageprocessing unit 308 in a periodic and consistent manner. In suchexamples, image capture device 304 may capture images at a capture ratethat corresponds to an exposure determination rate of controller 306.The term “exposure determination rate” refers to a rate at whichcaptured images are processed by the controller to determine sets ofexposure parameter values. For example, controller 306 can process acaptured image to determine a value for one or more of an exposureduration, gain, or aperture to use when capturing a subsequent image,and the exposure determination rate can refer to the rate at which suchprocessing occurs. More specifically, the exposure determination ratemay be the rate at which the controller determines pixel statistics forone or more captured images and determines sets of exposure parametersbased at least on the pixel statistics. The exposure determination ratemay be a multiple of the sampling rate, such that, for a plurality ofimages captured by image capture device 302, every nth image is sent toimage processing unit 308. For instance, the exposure determination ratecould be thirty or sixty frames per second, while the sampling ratecould be five or ten frames per second.

Controller 306 may determine the sets of one or more exposure parametervalues in accordance with the capture rate. For example, controller 306may receive a first image, determine pixel statistics for the firstimage, and determine a set of exposure parameter values for imagecapture device 304 to use when capturing a second image. Controller 306can determine the set of exposure parameter values for the second imagebased at least on the determined pixel statistics for the first image.For example, controller 306 can compare histogram data represented inthe pixel statistics to a desired histogram, and can alter one or moreexposure parameter values used to capture the first image such that thepixel statistics from the second image more closely match the desiredhistogram.

In other examples, the sampling rate can be a variable rate. Forexample, each time image processing unit 308 completes an imageprocessing operation, image processing unit can send a prompt tocontroller 306. Thus, controller 306 can receive a plurality of promptsthat correspond to a subset of the plurality of images. Controller 306can send the subset of images one at a time as the prompts are received.Accordingly, as time for performing an image processing operationchanges over time, so too will the sampling rate. In other examples,controller 306 or image processing unit 308 can alter the sampling ratebased on a number and/or type of image processing operations to beperformed on an image or images. For instance, image processing unit 308may receive a command to perform object detection operations for a firstperiod, and to perform deblur operations in addition to the objectdetection operations for a second period. Image processing unit 308 canset a first sampling rate during the first period and set a secondsampling rate during the second period, where each sampling ratecorresponds to an expected image processing time for the imageprocessing operations. Image processing unit 308 can send a feedbacksignal that indicates the different sampling rates, and controller 306can change the sampling rate based on the feedback signal. Otherexamples of variable sampling rates are possible as well.

One or more exposure parameter values can be altered similarly to thesampling rate. For instance, an exposure duration, gain, or apertureused to capture one or more images of the plurality of images can bealtered based on a feedback signal received from image processing unit308. In some examples, image processing unit 306 can determine that oneor more images sampled from the plurality are too bright or too dim, andcan send a feedback signal to controller 306 that indicates thatdifferent exposure parameters should be used for forthcoming images.Such feedback signals can be sent prospectively as well. For example,image processing unit 308 may determine a type of image processingoperation to be performed on forthcoming sampled images and can send afeedback signal that indicates desired characteristics for theforthcoming sampled images. For instance, where a color thresholdingoperation is forthcoming, image processing unit 308 can send a feedbacksignal indicating that a gain greater than one should be applied to afirst color channel, while a gain less than one should be applied to asecond color channel and a third color channel. Other examples offeedback from image processing unit 308 are possible as well.

In the present example, controller 306 and image processing unit 308 aredepicted as being separate, but in some examples they may be part of asingle unitary device, such as a video capture device, a camera, or animager. In some examples, image processing system 300 may be included asa component of a larger system, such as a robotic system. Separatingimage capture device 304 and controller 306 from image processing unit308 as depicted in FIG. 3 may reduce bandwidth usage within imageprocessing system 300 because not all of the images captured by imagecapture device 304 are sent for processing. In particular, wherecontroller 306 communicates wirelessly with image processing system 308,separating controller 306 from image processing system 308 in thisfashion may allow for efficient use of available communication bandwidthby sending only a subset of available images.

FIG. 4 illustrates a configuration of another image processing system400, according example embodiments. Image processing system 400 isincorporated into a robotic system. The robotic system may be similar oridentical to system 100 depicted in FIG. 1 . As illustrated in FIG. 4 ,image capture system 302, described above with regard to FIG. 3 , isincluded within sensors(s) 110 described above with regard to FIG. 1 .Image processing unit 308 is similarly incorporated into processor(s)102. An additional component, pre-processing unit 402, is similarlyincorporated into processor(s) 102. For purposes of clarity, controller108 depicted in FIG. 1 has been re-labelled robot controller 108.

As described above with regard to FIG. 3 , image capture system 302includes image capture device 304 and controller 306. Image capturesystem 302 and image processing unit 308 may interact within imageprocessing system 400 in substantially the same way described above.However, additional factors may contribute to the operation of thesecomponents in image processing system 400. For instance, robotcontroller 108 can send commands to either or both of controller 306 andimage processing unit 308 that affect the sampling rate. In one suchexample, robot controller 108 can be configured to plan successiveactions of a robot, such as the robot described above with regard toFIG. 2 . The planned actions may require image processing unit 308 toperform different image processing operations. Robot controller 108 cansend to controller 106 a sequence of commands to change the samplingrate based on the planned actions and, in parallel, send imageprocessing unit 308 a sequence of commands to perform various imageprocessing operations at different times. In another example, robotcontroller 108 can send a sequence of commands to image processing unit308 only, image processing unit 308 can send a feedback signal tocontroller 306, and controller 306 can change the sampling rate based onthe feedback signal. Other ways of carrying out the image processingoperations are possible as well.

Controller 306 can alter the sampling rate in other ways as well. Forinstance, in a mapped environment, image processing unit 308 may beexpected to carry out different image processing operations in differentlocations within the environment. Accordingly, controller 306 can alterthe sampling rate based on a location and/or orientation of the robotwithin the environment. Controller 306 may receive such location ororientation information from robot controller 108, from another sensorof sensor(s) 110, or from another component of system 100 describedabove with regard to FIG. 1 .

Image processing system 400 also includes pre-processing unit 402.Pre-processing unit 402 can also be configured to alter which images aresent to image processing unit 308. For example, pre-processing unit 402can sample images from a plurality of images captured by image capturedevice 304. Pre-processing unit 402 can determine a white level or focuslevel for the sampled images. The term “white level” refers to theintensity of pixels in a captured image. The term “focus level” refersto an identified in-focus portion or out-of-focus portion of a capturedimage. Different white levels or focus levels may be more suitable thanothers for processing by the image processing unit 308. Accordingly,where controller 306 receives a prompt from image processing unit 308 tosend an image, controller 306 can select an image to send based on thewhite level or focus level determined by pre-processing unit 402. Thoughpre-processing unit 402 is depicted as being separate from controller306, it should be understood that, in some examples, pre-processing unit402 and controller 306 might be one and the same. That is, controller306 can determine a white level and/or a focus level for some or all ofthe plurality of images in addition to determining sets of one or moreexposure parameter values for the plurality of images.

FIG. 5 illustrates a timing diagram 500 for an image processing system,according to example embodiments. In FIG. 5 , image capture device 304,controller 306, and image processing unit 308 are configured to performdifferent operations at different times.

At time 502, controller 306 is configured to send a first image to imageprocessing unit 308. Though not depicted in FIG. 5 , it should beunderstood that image capture device 304 first captured the first image.In timing diagram 500, image capture device 304 captures images at aconstant, or substantially constant, rate. Image capture device 304captures a second image at time 504, a third image at time 510, and annth image at time 516.

After receiving the first image, image processing unit 308 beginsperforming an image processing operation on the first image at time 506.As illustrated in timing diagram 500, the image processing time may takesignificantly longer than it takes controller 306 to determine a set ofone or more exposure parameter values. Controller 306 determines a setof exposure parameter values for the third image at time 508, anddetermines a set of exposure parameter values for a fourth image at time512 before image processing unit 308 completes the image processingoperation on the first image at time 514.

Once image processing unit 308 has completed the image processingoperation, it sends a feedback signal at time 518. In some examples, thefeedback signal simply includes a prompt to send another image. In otherexamples, the feedback signal can include an indication to change thesampling rate, an indication of a type of image forming operation to beperformed on another image, or an indication of target parameters forforthcoming images. For instance, image processing unit 308 candetermine that an image processing operation took too long because of animproper focus level, or that the operation was unsuccessful because animage was overexposed. In response, controller 306 can alter a thresholdfor the focus level, or constrain an error term threshold fordetermining the set of exposure parameter values to ensure thatforthcoming images are better suited for performance of image processingoperations.

After receiving the feedback signal, controller 306 sends an nth imageat time 520, and continues to determine a set of exposure parametervalues at time 522. In some examples, operation of the image processingsystem may continue in much the same way. In other examples, timingoperation of controller 306 and image processing unit 308 can changebased on external inputs.

Further, it should be understood that timing diagram 500 is a simplifiedillustration of communications between controller 306 and imageprocessing unit 308. Sending the images may further include exchangingacknowledgement signals between controller 306 and image processing unit308. Sending the images may also include modulating or otherwisetransforming image data. For instance, sending the image may includewirelessly sending the image data to the image processing unit accordingto a modulation scheme. In another example, controller 306 and imageprocessing unit 308 may send signals according to a controller areanetwork (CAN) protocol. Other such protocols and communication schemesare possible.

IV. Additional Embodiments

FIG. 6 illustrates operation of an image processing system 600,according to example embodiments. FIG. 6 illustrates an image capturedevice 304, a controller 306, and an image processing unit 308. FIG. 6also illustrates a plurality of images captured by the image capturedevice 304, which include images 602, 604, and 606, which are sampled bythe image processing unit. The depicted images are captured at differenttimes, such that image 602 is captured at a first time, image 604 iscaptured at a second time that is later than the first time, and image606 is captured at a third time that is later than the second time.

In some examples, image processing system 600 can be used in a roboticscontext. Image capture device 304 can capture a plurality of images at acapture rate commensurate with an exposure determination rate of thecontroller 306. Controller 306 can communicate with a robot controller,such as controller 108 described above with regard to FIG. 1 .Controller 306 can receive information related to one or more actions ofthe robot, and determine a sampling rate at which to send a subset ofthe plurality of images to image processing unit 308 based on the one ormore actions of the robot. Each of the one or more actions may berelated to an image processing performance time. For instance, an actionthat causes the robot to move may cause a large change in the scenecaptured by the image capture device. Image processing unit 308 mayperform more image processing operations due to the large change inscene, and may accordingly take a longer time to perform the imageprocessing operations. Accordingly, controller 306 may adjust thesampling rate based on the action of the robot. In other examples, imageprocessing unit 308 may send a feedback signal that indicates tocontroller 306 that the sampling rate should change. In FIG. 6 , everyseventh image is sampled, therefore, this example scenario may depict asystem where the sampling rate is constant, or where image processingunit 308 is taking a similar amount of time to perform operations ofeach of sampled images 602, 604, and 606.

In a robotics context, the sampling rate may be adjustable by controller306 based on a location of the robot. For instance certain imageprocessing operations may be associated with different locations in anenvironment of the robot. When a robot controller, such as controller108 described above determines a location of the robot has changed, itcan send signals to controller 306, and controller 306 can alter thesampling rate based on the command signals. Other examples are possiblein a robotics context as well.

Sets of exposure parameter values can be adjustable by controller 306 ina robotics context as well. For example, for a given set of exposureparameters, controller 306 can alter one or more of an exposureduration, gain, or aperture based on an action of the robot or based ona current location of the robot.

In further examples, image processing system 600 may be used in anautomotive context. That is, image capture device 304 and controller 306may be included within a vehicle, such as an autonomous vehicle. Imageprocessing unit 308 may be tasked with recognizing other vehicles in anenvironment of the vehicle, identifying symbols or text in road signs,perceiving pedestrians, animals, or other obstructions in theenvironment, detecting blur directionality associated with a suddenmovement of the vehicle, or other related image processing operations.Image processing unit 308 may be included within the vehicle, or may beremote from the vehicle. In examples where image processing unit 308 isremote from the vehicle, controller 306 may communicate with imageprocessing unit 308 wirelessly.

In an automotive context, controller 306 may interact with a vehiclecontroller. Similarly, controller 306 may interact with additionalsensors of the vehicle that provide controller 306 with informationrelevant to setting a sampling rate at which to send a subset of imagesto image processing unit 308. For example, proximity sensors of thevehicle may signal to controller 306 that an obstruction is within athreshold distance. Controller 306 can determine that image processingunit 308 will require more time to process each image based on thesignal, and can account for this by lowering the sampling rate. In otherexamples, a vehicle controller can send a control signal to controller306 based on a user input. Other examples in an automotive context arepossible as well.

In additional examples, system 600 can be used in a handheld devicecontext. In this example, image capture device 304 and controller 306can be included within a handheld device. Image processing unit 308 canbe included within the handheld device or be remote from the handhelddevice. Image processing operations on images captured by image capturedevice can include facial recognition operations, object detectionoperations, high dynamic range (HDR) operations, filtering operations,or others. Other examples in a handheld device context are possible aswell.

FIG. 7 illustrates operation of another image processing system 700,according to example embodiments. In FIG. 7 , system 700 includes imagecapture device 304, controller 306, and image processing unit 308.System 700 additionally includes pre-processing unit 402. Pre-processingunit 402 may be configured to perform image processing operations inparallel with image processing unit 308, and may help determine thesampling rate.

In some examples, one or more additional image capture devices may beused in addition to image capture device 304. For instance, imagecapture device 304 may be a first image capture device configured tocapture first images of a plurality of images at a first capture rate,and a second image capture device may capture second images at a secondcapture rate. Controller 306 may determine separate sets of one or moreexposure parameter values for each image capture device. Having twoimage capture devices simultaneously capture images can allow for afaster overall capture rate, which in turn allows controller 306 todetermine effective set of exposure parameter values more quickly. Insuch examples, sending a subset of the plurality of images may includeselecting the subset of images from the first images and the secondimages. The first images and second images can be captured according todifferent parameters, and so controller 306 may sample correspondingimages from the first image capture device and the second image capturedevice and send the corresponding images to image processing unit 308.For example, the first image capture device may focus on a first portionof a scene, the second image capture device may focus on a secondportion of the scene, and image processing unit 308 may be configured toperform HDR operations on corresponding images to form a combined imagethat is in focus at both portions of the scene.

In some examples, pre-processing unit 402 can be configured to determinea white level or focus level associated with a plurality of images.Image processing unit 308 may send a plurality of prompts to receiveimages from the plurality of images. In turn, controller 306 may selectan image from the plurality based on the determined white level and/orfocus level. In FIG. 7 , the plurality of images captured by imagecapture device 304 includes images 602, 702, 604, and 606. Of these,controller 306 samples and sends images 602, 702, and 606 to imageprocessing unit 308. These may be selected from other images of theplurality that are captured within a time threshold of each promptreceived from image processing unit 308. For example, images 702 and 604may be within the time threshold. In the present example, image 702 hasa more appropriate white level and/or focus level than image 604, and soimage 702 is sent to image processing unit 308.

Pre-processing unit 402 can additionally or alternatively determine alevel of change in the environment. For instance, pre-processing unit402 may determine a threshold percent change in a color (e.g. red,green, or blue) in a scene captured by two or more images of theplurality of images. Such a change may be indicative of additional imageprocessing operations for image processing unit 308 to perform.Pre-processing unit 402 may send a feedback signal to controller 306based on the level of change in the scene, and controller 306 can changethe sampling rate based on the feedback signal.

In other examples, pre-processing unit 402 or image processing unit 308can determine a number and type of image processing operations to beperformed on an image. Image processing unit 308 may identify fourseparate objects in image 602 and perform a deblur operation to theimage. Based on these operations, and a performance time associated withthe operations, image processing unit 308 can send a feedback signal tochange the sampling rate. The sampling rate can be adjusted to allowenough time to perform identical or similar image processing operationson the next sampled image. Pre-processing unit 402 may likewise performa preliminary check to determine which types of image processingoperations image processing unit 308 will likely carry out once an imageis sampled. Controller 306 can accordingly change the sampling ratebefore image processing unit 308 receives the image.

Though some of these example embodiments are discussed in a roboticscontext, an automotive context, or a handheld device context, it shouldbe understood that any embodiments described herein can be applied inother contexts as well. Further, though image capture device 304,controller 306, image processing unit 308, and pre-processing unit 402are depicted as being separate, it should be understood that any or allof these components might be included within a single unitary device.Still further, it should be understood that any of these embodiments maybe carried out as a method, or may be effectuated using a non-transitorycomputer readable medium having instructions stored thereon, that whenexecuted by one or more processors, perform a series of functions.

V. Example Operations

FIG. 8 illustrates an example block diagram of a method 800, accordingto an example embodiment. In some examples, method 800 may be carriedout as part of a system. For example, blocks 802, 804, and 806 may becarried out as part of the system described above with regard to FIG. 1, FIG. 2 , or FIG. 3 , FIG. 4 , FIG. 4 , and FIG. 5 . These steps may becarried out by the system components described above in conjunction withone or more processors executing program instructions stored on anon-transitory computer readable medium.

In other examples, the method may be carried out as part of a computingsystem. In these examples, a non-transitory computer readable medium maystore instructions executable by one or more processors to cause thecomputing system to perform the blocks of the method.

In these examples, the one or more processors and non-transitorycomputer readable medium may perform the blocks remotely. In otherexamples, the one or more processors and non-transitory computerreadable medium may carry out the method at a robot, vehicle, handhelddevice, or in another system context. In still other examples, portionsof the method may be carried out remotely, while other portions may becarried out at the robot, vehicle, handheld device, or other systemcontext.

Block 802 of the method 800 may be performed to determine, by acontroller of an image capture system, a plurality of sets of exposureparameter values for one or more exposure parameters. The plurality ofsets of exposure parameter values may be determined at an exposuredetermination rate, where the exposure determination rate includes arate at which captured images are processed by the controller todetermine each set of exposure parameter values.

Block 804 of method 800 may be performed to capture, by an image capturedevice of the image capture system, a plurality of images. Each image ofthe plurality of images may be captured according to a set of exposureparameter values of the plurality of sets of exposure parameter values.

Block 806 of method 800 may be performed to send, by the controller ofthe image capture system to an image processing unit, a subset of theplurality of images. The subset of images may be sent at a samplingrate, and the sampling rate may be less than the exposure determinationrate. In some examples, sending the subset of the plurality of images atthe sampling rate can include wirelessly sending the subset of images tothe image processing unit.

In some examples, the exposure determination rate may be a multiple ofthe sampling rate. In these examples, a controller, such as controller306 can send every nth image of the plurality of images to an imageprocessing unit, such as image processing unit 308. In other examples,sending the subset of images at the sampling rate can include receiving,by the controller of the image processing system from the imageprocessing unit, a plurality of prompts to send an image of the subsetof the plurality of images. In these examples, each image of the subsetmay be sent in response to a received prompt. In other examples, method800 may include receiving at a first time, by the controller of theimage capture system from the image processing unit, a first prompt tosend a first image of the plurality of images, receiving at a secondtime, by the controller of the image capture system from the imageprocessing unit, a second prompt to send a second image of the pluralityof images, and changing, by the controller of the image capture system,the sampling rate based on a difference between the first time and thesecond time.

In other examples, the image processing unit can provide feedbacksignals to the controller. For instance, method 800 can further includereceiving, by the controller of the image capture system from the imageprocessing unit, a feedback signal, and changing, by the controller ofthe image capture system, the sampling rate based on the receivedfeedback signal.

In additional examples, the image capture system can be included withina robot. In these examples, the image processing unit may be configuredto process images for a control system of the robot. In such examples,the method may also include the controller of the image capture systemchanging the sampling rate based on an action (e.g., a planned futureaction) of the robot.

Method 800 may include performing additional operations on the subset ofimages. For example method 800 may further the controller compressingthe subset of images prior to sending the subset of the plurality ofimages. In other examples, method 800 may further include determining,by the controller of the image capture system, a plurality of exposureerror terms that correspond to the plurality of images. In theseexamples, sending the subset of the plurality of images may be based onthe determined plurality of exposure error terms. The controller maydetermine whether an error term falls within an error term threshold. Ifthe error term falls within the error term threshold, the controller maysend an image of the plurality of images that corresponds to the errorterm to the image processing unit. In still other examples, method 800can also include determining a white level or a focus level of theplurality of images. In these examples, sending the subset of images mayinclude selecting the subset of images from the plurality of imagesbased on the determined white level or focus level.

Method 800 can also include altering one or more sets of exposureparameter values. For instance, method 800 can further includereceiving, by the controller of the image capture system from the imageprocessing unit, a feedback signal, and changing, by the controller ofthe image capture system, one or more sets of exposure parameter valuesof the plurality of sets of exposure parameter values based on thereceived feedback signal.

VI. Conclusion

The present disclosure is not to be limited in terms of the particularembodiments described in this application, which are intended asillustrations of various aspects. Many modifications and variations canbe made without departing from its spirit and scope, as will be apparentto those skilled in the art. Functionally equivalent methods andapparatuses within the scope of the disclosure, in addition to thoseenumerated herein, will be apparent to those skilled in the art from theforegoing descriptions. Such modifications and variations are intendedto fall within the scope of the appended claims.

The above detailed description describes various features and functionsof the disclosed systems, devices, and methods with reference to theaccompanying figures. In the figures, similar symbols typically identifysimilar components, unless context dictates otherwise.

The example embodiments described herein and in the figures are notmeant to be limiting. Other embodiments can be utilized, and otherchanges can be made, without departing from the spirit or scope of thesubject matter presented herein. It will be readily understood that theaspects of the present disclosure, as generally described herein, andillustrated in the figures, can be arranged, substituted, combined,separated, and designed in a wide variety of different configurations,all of which are explicitly contemplated herein.

A block that represents a processing of information may correspond tocircuitry that can be configured to perform the specific logicalfunctions of a herein-described method or technique. Alternatively oradditionally, a block that represents a processing of information maycorrespond to a module, a segment, or a portion of program code(including related data). The program code may include one or moreinstructions executable by a processor for implementing specific logicalfunctions or actions in the method or technique. The program code and/orrelated data may be stored on any type of computer readable medium suchas a storage device including a disk or hard drive or other storagemedium.

The computer readable medium may also include non-transitory computerreadable media such as computer-readable media that stores data forshort periods of time like register memory, processor cache, and randomaccess memory (RAM). The computer readable media may also includenon-transitory computer readable media that stores program code and/ordata for longer periods of time, such as secondary or persistent longterm storage, like read only memory (ROM), optical or magnetic disks,compact-disc read only memory (CD-ROM), for example. The computerreadable media may also be any other volatile or non-volatile storagesystems. A computer readable medium may be considered a computerreadable storage medium, for example, or a tangible storage device.

Moreover, a block that represents one or more information transmissionsmay correspond to information transmissions between software and/orhardware modules in the same physical device. However, other informationtransmissions may be between software modules and/or hardware modules indifferent physical devices.

The particular arrangements shown in the figures should not be viewed aslimiting. It should be understood that other embodiments can includemore or less of each element shown in a given figure. Further, some ofthe illustrated elements can be combined or omitted. Yet further, anexample embodiment can include elements that are not illustrated in thefigures.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopebeing indicated by the following claims.

What is claimed is:
 1. A method comprising: determining, by a controllerof an image capture system, a plurality of sets of exposure parametervalues for one or more exposure parameters; capturing, by an imagecapture device of the image capture system, a plurality of images,wherein each image of the plurality of images is captured according to aset of exposure parameter values of the plurality of sets of exposureparameter values; and based on the plurality of captured images,altering, by the controller of the image capture system, a sampling rateat which images are sent by the controller of the image capture systemto an image processing unit, wherein the image processing unit comprisesone or more processors configured to process the images.
 2. The methodof claim 1, wherein altering the sampling rate at which images are sentby the controller of the image capture system to the image processingunit comprises delaying sending the at least one of the plurality ofimages to the image processing.
 3. The method of claim 1, wherein eachset of the plurality of sets of exposure parameter values comprises aduration value and a gain value.
 4. The method of claim 1, furthercomprising: receiving, by the controller of the image capture systemfrom the image processing unit, a plurality of prompts to send an imageof the plurality of images, wherein each image of the plurality ofimages is sent in response to a received prompt.
 5. The method of claim1, further comprising: receiving, by the controller of the image capturesystem from the image processing unit, a feedback signal; and changing,by the controller of the image capture system, the sampling rate basedon the received feedback signal.
 6. The method of claim 1, furthercomprising: receiving, by the controller of the image capture systemfrom the image processing unit, a feedback signal; and changing, by thecontroller of the image capture system, one or more sets of exposureparameter values of the plurality of sets of exposure parameters basedon the received feedback signal.
 7. The method of claim 1, wherein theimages are wirelessly sent by the controller of the image capture systemto the image processing unit.
 8. The method of claim 1, wherein theimages are compressed before being sent by the controller of the imagecapture system to the image processing unit.
 9. The method of claim 1,further comprising determining a white level of the plurality of images,wherein images are selected for sending by the controller of the imagecapture system to the image processing unit based on the determinedwhite level.
 10. The method of claim 1, further comprising determining afocus level of the plurality of images, wherein images are selected forsending by the controller of the image capture system to the imageprocessing unit based on the determined focus level.
 11. The method ofclaim 1, wherein altering, by the controller of the image capturesystem, the sampling rate at which images are sent by the controller ofthe image capture system to the image processing unit comprisesdecreasing the sampling rate.
 12. The method of claim 11, furthercomprising subsequently increasing the sampling rate based on one ormore subsequently captured images.
 13. A system comprising: at least oneimage capture device; an image processing unit comprising to at leastone processor configured to process images; a controller comprising oneor more processors; a non-transitory computer readable medium; andprogram instructions stored on the non-transitory computer readablemedium and executable by the one or more processors of the controllerto: determine a plurality of sets of exposure parameter values for oneor more exposure parameters; cause the at least one image capture deviceto capture a plurality of images, wherein each image of the plurality ofimages is captured according to a set of exposure parameter values ofthe plurality of sets of exposure parameter values; and based on theplurality of captured images, alter a sampling rate at which images aresent by the controller to the image processing unit.
 14. The system ofclaim 13, wherein the images are wirelessly sent by the controller tothe image processing unit.
 15. The system of claim 13, wherein theimages are compressed before being sent by the controller to the imageprocessing unit.
 16. A non-transitory computer readable medium havingstored therein instructions executable by one or more processors tocause a computing system to perform functions comprising: determining,by a controller of an image capture system, a plurality of sets ofexposure parameter values for one or more exposure parameters;capturing, by an image capture device of the image capture system, aplurality of images, wherein each image of the plurality of images iscaptured according to a set of exposure parameter values of theplurality of sets of exposure parameter values; and based on theplurality of captured images, altering, by the controller of the imagecapture system, a sampling rate at which images are sent by thecontroller of the image capture system to an image processing unit,wherein the image processing unit comprises one or more processorsconfigured to process the images.
 17. The non-transitory computerreadable medium of claim 16, wherein altering the sampling rate at whichimages are sent by the controller of the image capture system to theimage processing unit comprises delaying sending the at least one of theplurality of images to the image processing.
 18. The non-transitorycomputer readable medium of claim 16, wherein altering, by thecontroller of the image capture system, the sampling rate at whichimages are sent by the controller of the image capture system to theimage processing unit comprises decreasing the sampling rate.
 19. Thenon-transitory computer readable medium of claim 16, the functionsfurther comprising determining a white level of the plurality of images,wherein images are selected for sending by the controller of the imagecapture system to the image processing unit based on the determinedwhite level.
 20. The non-transitory computer readable medium of claim16, the functions further comprising determining a focus level of theplurality of images, wherein images are selected for sending by thecontroller of the image capture system to the image processing unitbased on the determined focus level.